Linux Physical Memory Analysis

We present a tool suite for analysis of physical memory usage within the Linux kernel environment. This tool suite can be used to collect and analyze how the physical memory within a Linuxenvironment is being used. Embedded subsystems are common in today's computer systems. These embedded subsystems range from the very simple to the very complex. In such embedded systems, memory is scarce and swap is non-existent. When adapting Linux for use in this environment, we needed to keep a close eye on physical memory usage. When working on such a subsystem, we experienced various out of memory situations, ranging from significant application performance problems due to thrashing of executable pages to the killing of selected processes by the automated out of memory handling of the kernel. After numerous Internet searches, we determined that there was no publicly available tool suite to analyze physical memory usage in real time on a running system using the Linux kernel. There are many tools, such as mpatrol [1] or memprof [2], which can be used for memory analysis. These tools focus on dynamically allocated virtual memory usage within a particular process. However our requirement was for systematic view physical memory usage across all usage types. A significant amount of this information can alo be found in the /proc filesystem provided by the kernel. These /proc entries can be used to obtain many statistics including several related to memory usage. For example, /proc//maps can be used to display a process' virtual memory map. Likewise, the contents of / proc//status can be used to retreive statistics about virtual memory usage as well as the Resident Set Size (RSS). In most cases, this process level detail is sufficient to analyze memory usage. In systems that do not have backing store, more details are often needed to analyze memory usage. For example, it's useful to know which pages in a process' address map are resident, not just how many. This information can be used to get some clues on the usage of a shared library. Also, since this information is process based, there is no way to tell if the resident pages are shared between processes. Yet this sharing of pages is a very important optimization performed by the kernel that can not be tracked with the existing tools. We researched the Linux VMM from [3] and started the process of developing a tool suite to analyze the use of physical memory within a subsystem. This included the usage of memory by the kernel and user space applications. This paper focuses on the analysis of user space applications. Kernel usage was determined using the slab information. FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association 23 The basic requirement of the tool suite is to be able to determine physical memory usage in real time of an operational system with minimal impact to that system. In addition, in the case where a kernel panic did occur and a dump was available, the tool suite should be able to extract the pertinent information from the dump and provide the same analysis capability based off of the dump. The design goals were to keep it simple, keep it small, and provide useful information in a very quick turn around time. The tool suite was divided into data collection and data analysis tools. This allows each tool to be customized to the particular environment in which it must run. It also allows for a separation of the functionality and staging of the deliveries. In fact the initial delivery of the tool set contained only the data collection tools. The data was analyzed manually within a spreadsheet. The analysis tools were added later to help automate the analysis process. As mentioned previously, the tool suite is split into a data collection tool and a data analysis tool. The following sections describe each tool independently. The data collection tool started out with two versions. The first version was a loadable kernel module that was installed into the kernel and provided a user interface to extract the data. It used the global internal kernel data structures of the VMM to collect the various information of virtual and physical memory usage. The internal VMM structures need to be collected as the virtual memory of a process is created at the request of the process, but the physical memory is only assigned when it has been accessed. Without the VMM structures, there is no way to determine which virtual pages have physical pages associated with them. The second version was a user space application that probed the kernel memory through the /dev/kmem device. This version is more difficult to build and maintain than the module as kernel structures are being used in user space. Also, the impact of the user application on the running system was larger than that of the kernel module. Lastly, there was some function that was not possible to implement with the user space application. Due to these factors, only the kernel module is support for data collection on a running system. A third version of the data collection is available for the utility as described in [4]. The tool suite provides a data collection module that can extend the utility when the form of the Linux kernel dump is used. This provides data in the same format as the kernel module so that the same analysis tools can be used. All versions of the collection tools collect the same information in the same format. This is required to be able to feed into the same analysis tool. The data starts out with the process information. This includes the process identifier (PID), the command used to start the process, the value of the pointer to the memory management (MM) structures, page fault numbers, and used time counters. These last numbers have recently been added. The analysis tool can parse them but does not currently perform any analysis on them. The page fault numbers have been added to get an indication of how many pages faults happen over a period of time. That metric can be used as an indication of approaching the point of thrashing. After the process information is dumped, the data is dumped in a set of three values. The first part of this trio is the virtual memory area (VMA). This provides the virtual start address, length, and various flags. The second part of this trio is a summary of the page table entries (PTEs) that are associated to that VMA. The third part of the trio is the detailed mappings from virtual page to physical page that is provided by the PTE. There is also a set of flags with this information. The output is in a terse format to minimize space and limit the performance and memory requirements to collect and store the data. It dumps this information for all Linux tasks, resulting in a requirement on the analysis tool to sort out thread from processes. This can be done using the MM pointers. Sample output of the collected data is provided later in this document and shown in Figure 1. FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association 24 The data analysis tool takes as input the output from the data collection tools and produces a variety of formatted output. In its simplest mode, it provides a comma separated value file that can be imported into a spreadsheet for further analysis manually. It can also provide graphic output, in the form of PostScript, of various metrics of physical and virtual memory usage. The data analyzer parses information in the following forms. Basic process information, available in either physical or virtual memory views as PostScript or comma separated values, which includes: total number of read-only and read/write pages assigned to a process, number of pages assigned to the process stack, number of pages assigned to global data areas (both .bss and .data), number of pages assigned to heap space, number of pages assigned to the application executable code, number of pages in shared library global data (.data and .bss), number of pages in shared library executable code; A view of the unique virtual memory areas (VMAs) within the process including virtual and physical sizes and the mapping of physical pages within the virtual space; A view of all unique executable entities (application code and shared libraries) coalesced across all processes that shows the virtual and physical sizes as well as the mapping of physical pages within the virtual space; For all unique executable entities (application code and shared libraries) a view of the count of physical pages for the executable (i.e., .text) and global data (i.e., .data and .bss) areas of each entity, available as a physical or virtual view as PostScript or comma separated values; Detection within a virtual memory area of single and multiple use pages (more on this later); The total number of physical pages used by user space applications; A dump of the physical page structures used within the kernel that are assigned to user space processes for further manual analysis. Sample output of the various analyzed data is provided later in this document. The kernel module supports both PPC and Intel processor architectures. It uses only architecture independent structures, interfaces, and macros, and should be easily ported to other processor architectures. The dump data extractor supports a PPC dump being analyzed on an Intel Linuxenvironment. There are some architecture dependent structures used in this environment and therefore would be harder to port. The data analysis tool supports Intel Linux and Intel Cygwin operating environments. It uses a simple text based selection mechanism and has no graphical user interface. It should port easily to other GNU C environments. The following sections provide sample invocations and output of the various portions of the tool suite. The primary collection tool is the kernel module. It can be installed using the insmod command. The output of the kernel module is extracted from a special character device. In the current implementation, this device is created using devfs. A device interface was used as the amount of data can be quite significant. It also minimized the complexity in the driver by not having to provide a simple character stream for the data. All FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association 25 of the buffering has been pushed out of the kernel and into user space. A simple mechanism to extract this information is to use the dd command. In order to keep the kernel module simple, the device actually blocks all of the task data into one read command. To determine the minimumread size, use the information in the following file. The content of this file will specify the maximumsize of an output block. Using this information, the following command can be used to extract the physical per process usage. Now the output will be located in . Since this information can be quite large, it is unlikely that the information will be able to be stored directly on the embedded system. In our usage, an NFS mount was used to store the output file. Sample output of the collected data is shown in Figure 1. This example provides a good sample of the data. The task line starts with the T: and provides information on the kernel task. This data includes the command use to start the task, the process identifier, the pointer to the memory map, the CPU number that this process is associated with, page fault information, and usage times. The memory map pointer is used by the analysis tools to determine the difference between a pthread within a process (with the same memory map) and a unique process with the same name (different memory map). The other information, such as the page faults, can be used to detect thrashing. The time measurements could be used by performance analyzers to determine the usage patterns across various processes across time. For each of the tasks, the virtual memory areas (VMAs) are listed and each starts with V:.Each VMA item shows the starting and ending virtual address of the area, the size in virtual, the size that is physically resident in memory, various flags, and the executable unit to which this VMA belongs. The various flags are critical to analyze the usage of the VMA. From the flags, it can be determined if the VMA is used for text, data, or stack areas. Also, the executable unit is used in the analysis. For ! " # # # # # # # # # $%&'& ! # (# () # # $%'*%+# (& # &%)*+# (+,*&%# )++(# $ (%$ -! ./%, # ..0 ...# ((1 # *1 # ,('# +*# ' ' * 2! %$# # %$# # %$ 3! ./%, ')+& ('# ''# ,' 44 5 ./%* '),% ('# ''# ,' 44 5 ./%& '),) ('# ''# ,' 44 5 ./%+ '),+ ('# ''# ,' 44 5 ./%) '),& ('# ''# ,' 44 5 ./%% '),* ('# ''# ,' 44 5 ./%6 '),, ('# ''# ,' 44 5 ./%0 '),$ ('# ''# ,' 44 5 -! ..0( # ..0$...# $ # # , '# + # ' ' * 2! # # # # 3! -! ..0, # ..4+...# (, # ) # ,('# ++# ' ' * 2! )# # *# )# 3! ..4 ')*+ %'# (# *, 5 ..4( ')*&,%'# (# 1, 5 ..4' ')**,%'# (# 1, 5 ..4$ '$/(*1'# (# ' 4, 5 ..4, ')+ &%'# (# ' 1, 5 ..4* ')*$,%'# (# 1, 5 ..4& '646*1'# (# ' 4, 5 ..4+ ''*.+1'# (# ' 4, 5 -! ..4) # ..44...# * # , # ,('# ++# 7899 2! ,# # $# ,# 3! ..4) ')& &%'# (# '' 1, 5 ..4% ')(%,%'# (# 1, 5 ..46 '+%' %'# (# *, 5 ..40 (1$/*1'# (# ' 4, 5 ! # '# # # # # # # # ''% ! :# $# # # # # # # # '$( ! # (%# () ' # # (,# +# # # +&'# (%+( -! .4$6 # .4,,...# 0 # $ # ,('# +*# ( 2! $# # $# # $ 3! .4$6 '*0$ ('# (*# ,' 44 5 .4$0 '*61'('# (*# ,' 44 5 .4,, '*0 '('# (*# ,' 44 5 -! .4,* # .4,%...# * # # , '# + # ( 2! # # # # 3! -! .4,6 # .4**...# 4 # ' # ,('# ++# ( 2! '# # '# '# 3! .4*, '*6* %'# (# * 5 .4** '*6/'%'# (# ' * 5 Figure 1 Sample Data Collector Output FREENIX Track: 2005 USENIX Annual Technical Conference USENIX Association 26 anonymous areas, the executable unit is NULL. This is typical for heap areas and bss areas. Also with the VMA is the page table entry summary for that particular VMA. This is started with the P: key. The information in this stanza of the output is the number of user space pages, executable pages, read-only pages, single-use pages, and shared pages. All of these are in terms of physically resident pages. For those VMAs with physically resident pages, the mapping from virtual to physical is also provided by the M: stanza. The information provided here includes the virtual address, the physical address, the usage count of this physical page, and various page flags. Note that these are separated with a unique character to ease parsing in the analyzer.